Use of multiple player real-time voice communications on a gaming device

ABSTRACT

A game console capable of communicating with other game consoles over a link or network is provided with a headphone and microphone for each player who will engage in voice communication. Verbal communications directed to one or more other players are converted to pulse code modulated (PCM) digital data and are encoded and compressed in real-time, producing data packets that are transmitted to another game console. The compressed data packets are decompressed and decoded, producing PCM data that are converted to an analog signal that drives a headphone of the intended recipient. Players can selectively mute voice communications to and from a specific other player. The PCM data can be encoded in a round-robin fashion that reduces the number of encoders required. A predefined level of computing resources is used for voice communication to avoid aversely affecting the quality of game play.

RELATED APPLICATIONS

This is a divisional application of U.S. Ser. No. 10/147,578, which wasfiled on May 16, 2002, now U.S. Pat. No. 6,935,959, and the benefit ofthe filing date thereof is hereby claimed under 35 U.S.C. § 120.

FIELD OF THE INVENTION

The present invention generally relates to communication between playersof an electronic game; and more specifically, pertains to a multiplayerelectronic game system that facilitates voice communication betweenplayers using one or more multiplayer electronic gaming devices,including voice communication over a network that conveys data betweenthe multiplayer electronic gaming devices that are coupled together toenable the player to participate in a game.

BACKGROUND OF THE INVENTION

When playing a non-electronic game with one or more other people, forexample, a card game such as bridge, the social interaction arisingthrough verbal communication between the players during the gametypically adds much to the enjoyment of the game. Verbal communicationis also often an element of game play, since comments made by a playerto an opponent during a game can have the effect of causing the opponentto lose concentration and perform poorly, while comments made to teammembers can provide encouragement, thereby improving their quality ofplay. Verbal communication between persons playing games is thus clearlyan important element of the gaming experience.

The verbal repartee between players that is so important to game playwas initially missing when players first began to play electronic gamesover the Internet and other network links. Players at different siteswere generally not able to communicate with each other, because theirpersonal computers (PCs) only communicated data related to the play of agame over the network. The loss of the verbal communication and relatedsocial interaction that is such an important aspect of games played bypeople at the same location thus caused games played over the Internetto be less interesting. To address this problem, hardware and softwaresolutions were developed that support voice communications between PCsover the Internet or other network during game play. At about the sametime, techniques were developed to convey voice over the Internet orother networks (i.e., voice over IP) to enable communications betweenparties connected by the network without incurring the cost ofconventional telephone long distance calls. This work resulted in thecreation of various protocols supporting voice over IP communication,including the H.323 specification, Session Initiation Protocol (SIP),and Media Gateway Control Protocol/Media Gateway Controller(MGCP/MEGACO) specification. The techniques developed for voice over IPare generally applicable to and have been used in schemes to enableverbal communications between players of PC electronic games over anetwork. Examples of systems that provide voice communication betweenconnected PCs during game play include Microsoft Corporation's SIDEWINDER GAME VOICE™, Mindmaker, Inc.'s GAME COMMANDER 2™ device andsoftware, TEAMSOUND™ software, GameSpy Industries' ROGER WILCO™software, and Alienware Technology's FIRST CONTAC™ software. The voicecommunication provided by these products greatly adds to the enjoymentof playing games on PCs that are connected over the Internet or othernetworks. Some of these systems operate in peer-to-peer mode, in whichvoice data are transferred over the network directly between PCs, whileothers require a voice server that receives the voice data from one gameplayer's PC and forwards the voice data over the network to one or moreother PCs that are connected to the network for playing the game.

Since these systems provide communication for only one player per PC,each PC produces its own network stream of voice data, and this networkstream is directed to other PC of each other player (or to the voiceserver, which then directs the voice data to the PC of each otherplayer). This approach thus produces a substantial network dataoverhead.

Currently, none of the prior art systems for PC game voice communicationenable multiple players per PC in a game played over the Internet orother network and therefore, the prior art does not supportmultiplayer-per-PC voice communication functionality. Also, if suchmultiplayer PC systems were developed using existing voice communicationprotocols for several players on a PC, they would likely require anexcessive amount of computational resources. Allocating the requiredresources to voice communication for all players of the game on a PCmight well have an adverse effect on the quality of game play, unlessthe PC had a very fast processor, lots of memory, and a fast videodriver card.

In contrast to PCs, dedicated game consoles often do not have theprocessing power and available memory of a powerful PC, so this problemis of even greater concern in developing a scheme to support voicecommunications by multiple players on each game console. It would bedesirable to allocate a fixed level to the requirements for memory andother computer resources needed for voice communication, independent ofthe number of players who are capable of voice communication on the gameconsole, as appropriate for the game functionality and design, so thatthe resources required for voice processing are not allowed to increasebeyond a defined limit as the number of players participating in voicecommunication changes. It would also be advantageous to enable voicecommunications between multiple players for each instance of a game at asite, and to enable each player to control with whom the player verballycommunicates (both speaking and listening), and to combine all of thevoice data from that instance of the game into a single network datastream in order to reduce the network bandwidth required for voicecommunications between multiplayer game consoles. It would further bedesirable to share certain resources, such as a voice data encoder ordecoder, between multiple players for a single game instance at a site.

As the quality of game graphics improves, it becomes more important tomaintain other features relating to realism. One such feature is theability to provide lip sync or other viseme (lip position) informationwith the voice data during game play, to enable the lips of a graphiccharacter displayed in a game to move in synchronization with the wordsof a player who is represented by and controlling the graphic characterin the game display. However, existing voice communication systemstypically do not convey data to enable lip sync, and as a result, theplayer receiving the voice communication will not see the lips of thecharacter in the game corresponding to the player who is speaking movein sync with the speaker's words.

While voice communication is generally a desirable feature, if misusedby a specific player, it may become annoying and reduce the enjoyment ofgames by other players. The prior art also does not enable a player toblock an annoying player from talking or listening to the player duringany game in which an annoying player is a participant, regardless of thegame console used by the annoying player. Different people have varyingdegrees of tolerance for annoying behavior of others. However, anyplayer who chooses not to listen or speak to a specific other player forany reason, must have the ability to prevent communication with thespecific other player, without giving up the ability to communicate withother players in games. A player may be upset because the player feels aspecific other player uses excessive profanity or sexually explicitlanguage, or uses language or makes comments that the player feels to bederogatory or socially unacceptable.

A parent may also want to block a child from participating in voicecommunication during game play to avoid exposing the child to anyprofanity and to preclude verbal communication with someone who mightattempt to contact the child outside the scope of game play for harmfulpurposes. This parental block of a child's voice communication should bestored on an online game service so that it remains in effect if thechild connects to the online game service from a different game console.The prior art game voice communication systems do not permit blockingverbal communications by a selected player, such as a child,participating in games using a multiplayer game console.

Should any player's verbal conduct while playing games over the Internetor other network become so egregious (based upon the number ofcomplaints received from other players) as to warrant it, an online gameservice should be able to prevent that player from participating invoice communication while playing games through the online game servicefor a period of time, and if further justified by the continued receiptof complaints about that player's verbal behavior, to ban the playerfrom using voice communication permanently. The current voicecommunication systems do not enable this level of control to be appliedat each online game service so that a player is banned regardless of analias used or the game console through which the player participates ingame play through the online game service.

Although voice communications systems are well known for use on PCsplaying games, gaming consoles have a different architecture, with adefined limit on available system resources. Most game consoles enablemultiple players on a single instance of a game, i.e., on the sameconsole. When gaming consoles are interconnected over the Internet orother network, they should preferably be able to provide voicecommunications for each of multiple players on the game console. Use ofconventional techniques that have been developed to enable voicecommunication during game play on PCs will be unacceptable on gameconsoles, because of the more limited computing resources and the needto accommodate voice communications for multiple players. Accordingly,there is clearly a need for a method and system to enable voicecommunication for games played on one or more multiplayer game consoles,which addresses the issues noted above.

SUMMARY OF THE INVENTION

As noted above, voice communication between one person playing a game ona PC connected over a network and one or more other players of the game,each using a separate PC is well known. In contrast, the presentinvention enables more than one person playing a game on a game consoleto verbally communicate locally and/or over a network with one or moreother people playing the game. To avoid creating too much of a demandfor computing resources to support verbal communication as more peoplejoin in the game, the present invention enables a designer of the gameto establish a predefined limit on the computing resources that will beemployed to support verbal communication between players of the game.

Specific hardware is provided for use with each multiplayer game consoleto support verbal communication between players of a game. Each playerwho is verbally communicating is provided with a headset that includes amicrophone (more generally, an audio sensor) and a headphone (moregenerally, an audio transducer). The headset is coupled to a voicecommunication module, which is either attached to a game controller inone embodiment, or integrally included in a game controller in anotherembodiment. An input signal from the microphone is compressed anddigitized by an encoder in the multiplayer game console. The digitizedand compressed signal is then conveyed through a voice communicationchannel to another player who is an intended recipient. A decoderdecompresses the compressed signal to produce an output signal that isapplied to the headphone of the other player, to produce an audiblesound corresponding to the sound originally produced and incident on themicrophone of the player from whom the voice communication was conveyed.As used herein and in the claims that follow the term “voicecommunication” is intended to be synonymous and interchangeable with theterm “verbal communication.” Also, it will be understood that “verbalcommunication” or “voice communication” are intended to encompass theconveyance of spoken words and phrases, as well as other sounds orutterances such as grunts, screams, etc. that are produced by a playerparticipating in a game.

A level of computing resources allocated to processing verbalcommunications on the multiplayer game console is preferably predefinedand fixed so that it is independent of the number of players using voicecommunication while playing the game. This feature ensures that anincrease in the number of players who are verbally communicating duringa game does not adversely impact upon the quality of other aspects ofthe game play.

Data comprising the digital compressed signal are in a specific formatwhen conveyed to one or more intended recipients playing the game on oneor more game consoles. The digital compressed signal is conveyed overeither a direct link or through a network, such as the Internet.Depending upon the game, each player can be enabled to select anintended recipient of a verbal communication from among the otherplayers of the game. For example, members of a team can verballycommunicate with each other and with a team leader over one voicechannel, but the team leader may be able to selectively communicate withthe team members over that same voice channel or with another teamleader over a different voice channel.

Preferably, the step of encoding includes the step of converting theinput signal from an analog signal to a digital signal, and compressingthe digital signal to produce a compressed digital signal in theappropriate format. Similarly, the step of decoding preferably includesthe steps of decompressing the compressed signal to produce adecompressed digital signal, and converting the decompressed digitalsignal to the output signal used to drive the audio transducer.

The format used for the compressed signal comprises a plurality of audiodata packets. Each audio data packet extends over an audio time frame.Preferably, a predefined number of encoding instances are operative onthe game console during each audio time frame. The predefined number ispreferably less than a total number of players using the game consoleand producing verbal utterances to verbally communicate with otherplayers. The verbal utterances of players on the game console areencoded in such a manner that if more players on the game console thanthe predefined number of encoding instances are speaking in successiveaudio time frames, a round robin selection is applied in choosing theverbal utterances that are encoded in successive audio time frames. Thegame being played determines the predefined number of encoding instancesactive at one time.

Parallel decoding of encoded data streams received from othermultiplayer game consoles can be employed in conjunction with one ormore mixers to produce the output signal that drives the players'headphones.

Selected players can be assigned to one or more channels, and playerswho are assigned to a common channel are enabled to selectively verballycommunicate with each other. In one embodiment, each player is assignedto one or more listener channels on which the player can receive averbal communication from other players. In addition, each player isassigned to one or more talker channels over which a verbal utterance isconveyed to other players. The verbal utterances heard by a specificplayer on the multiplayer game console are determined by logicallycombining (i.e., ANDing) the listener channel of the specific playerwith the talker channel of the player making the verbal utterance.

The method also includes the step of providing oral synchronization datafor controlling an oral portion (e.g., the mouth) of an animated graphiccharacter in the game. The oral synchronization data are used to controlthe oral portion of the animated graphic character that is displayedduring the game, so that it moves in synchronization with the verbalutterance of the player who controls the animated graphic character.

A player is enabled to prevent verbal communication with a selectedother player. Verbal communication with the selected player can beprevented in just a current game being played, or can be prevented inany game in which both the player and the selected other player areparticipating.

A further aspect of the present invention is directed to a system thatenables verbal communication between players who are playing a game. Thesystem includes a multiplayer game console having a processor and amemory. In the memory are stored machine instructions for causing theprocessor to carry out a plurality of functions, including executing aninstance of a game. Verbal communication input and output devices areincluded for each player who will be verbally communicating during agame, and each verbal communication input and output device has a soundsensor that produces an input signal indicative of sound incident on thesound sensor, and a sound transducer that produces an audible sound inresponse to an output signal that is applied to the sound transducer.The machine instructions in the memory cause the processor to carry outfunctions generally consistent with the steps of the method describedabove to enable verbal communication.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

The foregoing aspects and many of the attendant advantages of thisinvention will become more readily appreciated as the same becomesbetter understood by reference to the following detailed description,when taken in conjunction with the accompanying drawings, wherein:

FIG. 1 is a schematic isometric view of a multiplayer game console andvoice communication system in accord with the present invention;

FIG. 2 is a block diagram of the multiplayer game console and voicecommunication module of FIG. 1;

FIG. 3 is a functional block diagram of a multiplayer game console withvoice communication capability;

FIG. 4 is a functional block diagram illustrating two multiplayer gameconsoles coupled in point-to-point communication over a network;

FIG. 5 is a block diagram illustrating a first multiplayer game consolecoupled in communication with three other multiplayer game consoles overa network;

FIG. 6 is functional block diagram illustrating prioritization encodingfor a plurality of players on a game console having two parallelencoders;

FIG. 7 is a logic diagram illustrating the steps employed by the presentinvention in selecting packets from a queue to decode on a multiplayergame console;

FIG. 8 is a functional block diagram illustrating a Type 1 decodingengine used in the multiplayer game console;

FIG. 9A is a functional block diagram illustrating a Type 2 decodingengine used in the multiplayer game console;

FIG. 9B is a block diagram illustrating details of the mixers and4-stream parallel decoder of FIG. 9A;

FIG. 10 is a logic diagram illustrating further details of the steps fortransmitting and receiving encoded packets of sound data over a network;

FIG. 11 is a functional block diagram showing how voice streams arereceived, queued, and decoded for each player on a multiplayer gameconsole;

FIG. 12 is a flow chart illustrating the steps carried out for roundrobin encoding of sound packets;

FIG. 13 is an exemplary user interface for selecting voice options in agame that employs voice communication between players; and

FIGS. 13A and 13B illustrate two different options that are selectableby a player to control or preclude voice communication with anotherplayer in a multiplayer game, in accord with the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENT

Exemplary Gaming System for Practicing the Present Invention

As shown in FIG. 1, an exemplary electronic gaming system 100 includes agame console 102 and support for up to four user input devices, such ascontrollers 104 a and 104 b. Game console 102 is equipped with aninternal hard disk drive (not shown in this Figure) and a portable mediadrive 106 that supports various forms of portable optical storage media,as represented by an optical storage disc 108. Examples of suitableportable storage media include DVD discs and compact disk-read onlymemory (CD-ROM) discs. In this gaming system, game programs arepreferably distributed for use with the game console on DVD discs, butit is also contemplated that other storage media might instead be used,or that games and other programs can be downloaded over the Internet orother network.

On a front face of game console 102 are four connectors 110 that areprovided for electrically connecting to the controllers. It iscontemplated that other types of connectors or wireless connectionsmight alternatively be employed. A power button 112 and a disc trayeject button 114 are also positioned on the front face of game console102. Power button 112 controls application of electrical power to thegame console, and eject button 114 alternately opens and closes a tray(not shown) of portable media drive 106 to enable insertion andextraction of storage disc 108 so that the digital data on it can beread and loaded into memory or stored on the hard drive for use by thegame console.

Game console 102 connects to a television or other display monitor orscreen (not shown) via audio/visual (A/V) interface cables 120. A powercable plug 122 conveys electrical power to the game console whenconnected to a conventional alternating current line source (not shown).Game console 102 may be further provided with a data connector 124 totransfer data through an Ethernet connection to a network and/or theInternet, or through a broadband connection. Alternatively, it iscontemplated that a modem (not shown) may be employed to transfer datato a network and/or the Internet. As yet a further alternative, the gameconsole can be directly linked to another game console via an Ethernetcross-over cable (not shown).

Each controller 104 a and 104 b is coupled to game console 102 via alead (or in another contemplated embodiment, alternatively, through awireless interface). In the illustrated implementation, the controllersare Universal Serial Bus (USB) compatible and are connected to gameconsole 102 via USB cables 130. Game console 102 may be equipped withany of a wide variety of user devices for interacting with andcontrolling the game software. As illustrated in FIG. 1, each controller104 a and 104 b is equipped with two thumb sticks 132 a and 132 b, aD-pad 134, buttons 136, and two triggers 138. These controllers aremerely representative, and other gaming input and control mechanisms maybe substituted for or used in addition to those shown in FIG. 1, forcontrolling game console 102.

Removable function units or modules can optionally be inserted intocontrollers 104 to provide additional functionality. For example, aportable memory unit (not shown) enables users to store game parametersand port them for play on another game console by inserting the portablememory unit into a controller on the other console. Other removablefunction units are available for use with the controller. In connectionwith the present invention, a removable function unit comprising a voicecommunicator module 140 is employed to enable a user to verballycommunicate with other users locally and/or over a network. Connected tovoice communicator module 140 is a headset 142, which preferablyincludes a boom microphone 144 or other type of audio sensor thatproduces an input signal in response to incident sound, and an headphone146 or other type of audio transducer for producing audible sound inresponse to an output signal from the game console. In anotherembodiment that is being contemplated (not shown), the voicecommunicator capability is included as an integral part of a controller(not shown) that is generally like controllers 104 a and 104 b in otherrespects. The controllers illustrated in FIG. 1 are configured toaccommodate two removable function units or modules, although more orfewer than two modules may instead be employed.

Gaming system 100 is of course capable of playing games, but can alsoplay music, and videos on CDs and DVDs. It is contemplated that otherfunctions can be implemented by the game controller using digital datastored on the hard disk drive or read from optical storage disc 108 indrive 106, or from an online source, or from a function unit or module.

Functional Components for Practicing the Present Invention

Turning now to FIG. 2, a functional block diagram illustrates, in anexemplary manner, how components are provided to facilitate voice orverbal communication between players during the play of electronic gameson the multiplayer game console. As noted above, this embodiment of gameconsole 100 can have up to four players on each console, and each playercan be provided with a controller and voice communicator. Details of avoice communicator module 140′ are illustrated in connection with itsassociated controller 104 a. It will be understood that controllers 104b, 104 c, and 104 d (if coupled to game console 100) can optionally eachinclude a corresponding voice communication module 140′ like thatcoupled to controller 104 a. In a current preferred embodiment, voicecommunication module 140′ includes a digital signal processor (DSP) 156,an analog-to-digital converter (ADC) 158, a digital-to-analog converter(DAC) 161, and a universal serial bus (USB) interface 163. In responseto sound in the environment that is incident upon it, microphone 144produces an analog output signal that is input to ADC 158, whichconverts the analog signal into a corresponding digital signal. Thedigital signal from ADC 158 is input to DSP 156 for further processing,and the output of the DSP is applied to. USB interface 163 forconnection into controller 104 a. In this embodiment, voicecommunication module 140′ connects into the functional unit or moduleport on controller 104 a through a USB connection (not separatelyshown). Similarly, digital sound data coming from game console 100 areconveyed through controller 104 a and applied to USB interface 163,which conveys the digital signal to DSP 156 and onto DAC 161. DAC 161converts the digital signal into a corresponding analog signal that isused to drive headphone 146.

With reference to multiplayer game console 100, several key functionalcomponents are shown, although it should be understood that otherfunctional components relevant to the present invention are alsoincluded, but not shown. Specifically, game console 100 includes acentral processing unit (CPU) 150, a memory 152 that includes both readonly memory (ROM) and random access memory (RAM). Also provided is a DSP154. The digital signal produced by ADC 158 in response to the analogsignal from microphone 144 is conveyed through controller 104 a to CPU150, which handles encoding of the voice stream signal for transmissionto other local voice communication modules and to other game consolesover a broadband connection through an Ethernet port (not shown in FIG.2) on the game console.

An alternative embodiment employs DSP 156 in voice communication module140′ to encode the digital signal produced by ADC 158 in response to theanalog signal from microphone 144. The encoded data are then conveyedthrough controller 104 a to CPU 150, which again handles transmission ofthe encoded data to other local voice communication modules and othergame consoles over the broadband connection on the game console.

It should be noted that multiplayer game console 100 can be eitherdirectly connected to another game console using a crossover Ethernetcable as a link, or can be connected to one or more other multiplayergame consoles through a more conventional network using a hub, switch,or other similar device, and/or can be connected to the Internet orother network through an appropriate cable modem, digital subscriberline (DSL) connection, or other appropriate interface broadbandconnection. An alternative embodiment is also contemplated in whichmultiplayer game console 100 is connected to the Internet or othernetwork through a modem (not shown). Digital signals conveyed as packetsover a direct or network connection are input to CPU 150 through theEthernet port on game console 100 (or from other voice communicationmodules and controllers connected to the same game console), and areprocessed by the CPU to decode data packets to recover digital sounddata that is applied to DSP 154 for output mixing. The signal from DSP154 is conveyed to the intended voice communication module for theplayer who is the recipient of the voice communication for input throughUSB interface 163.

An alternative embodiment employs the CPU to convey the encoded datapackets to intended voice communication module 140′ through controller104 a. The encoded data packets are then decoded by DSP 156 in voicecommunication module 140′, and the resulting decoded signal is conveyedto DAC 161, which creates a corresponding analog signal to driveheadphone 146.

In still another contemplated alternative, the headphone and microphonefor each player can be coupled directly to the game console and thefunctions of the voice communication module can be carried out by theCPU or other processor such as a DSP, and appropriate DAC and ADCmodules in the game console. The location of the components that processsound signals to produce sound data conveyed between players and toproduce the analog signals that drive the headphone of each player isthus not critical to the present invention.

CPU 150 also applies voice effects to alter the characteristics of thesound of a player speaking into microphone 144 and is able to change thecharacter of the sound with a selection of different effects. Forexample, a female player can choose a voice effect to cause her voice tosound like the deep-tone voice of a male, or so that the voice has anelfin quality, or so that it has one of several other desired differenttonal and pitch characteristics. Available voice effects from which aplayer can choose are game dependent. Such voice effects cansubstantially alter the sound of the player's voice so that the playeris virtually unrecognizable, and can add drama or greater realism to acharacter in a game being controlled by a player, when the characterappears to speak to other characters in the game. The voice effects thusfacilitate role playing and mask the player's true identity. Even whenplayers connected to the same game console 100 are directly audible toeach other because they are only a few feet apart in the room in whichthe game console is disposed, the change in a player's voice due tovoice effects being applied so alters the sound heard by other playersreceiving the verbal communication through their headphones that thelocal sound of a player's voice propagating within the room to theplayers can easily be ignored.

While not actually a limitation in the present invention, a currentpreferred embodiment of the game console 100 is designed with theexpectation that a maximum of up to 16 players can engage in verbalcommunication during a game being played over a network or over theInternet through an online game service. Clearly, there is a practicallimit to the number of verbal communications from other players withwhich a player might expect to engage at one time. Accordingly, it wasassumed that a player is unable to comprehend verbal communications frommore than four other players speaking simultaneously.

Game Play Scenarios

There are different appropriate scenarios, depending upon the type ofgame and the number of players engaged in a given game that affect therequirements for encoding and decoding voice communication signals. Forexample, there are three primary scenarios that impact on therequirements for voice communication. The first scenario is referred toas “point-to-point” and includes one player on each of twointerconnected game consoles, where each player is engaged in voicecommunication with the other player. In the second scenario, which isreferred to as “multipoint,” there is again only one player who isengaged in voice communication on each game console, but up to 16 gameconsoles are interconnected over a network for play of a game, in whichup to 16 players are participating. The third scenario is referred to as“multiplayer on game console,” since up to four players per game consoleand up to four game consoles can be interconnected over a network toenable up to 16 players to simultaneously play a game and verballycommunicate. In regard to the last scenario, two or more players on asingle game console can also use voice communication during a gamealthough they are physically located within the same room, since thebenefits of the voice changes produced by use of the voice effectsoption can enhance the enjoyment of the game and role playing by eachplayer, as noted above. Further, the limits of the total number of gameconsoles/player referenced above in each of the three scenarios can bethought of as soft limits, since there is no inherent hardwarelimitation precluding additional players or game consoles participating.

By designing games in accord with one or more of these three scenarios,it is possible for the software designer to set a maximum predefinedlimit on the computing resources that will be allocated to voicecommunication, to avoid voice communication from adversely impacting thequality of game play. Also, a specific game that is played on themultiplayer game console can have its own requirements so that it isappropriate for play by only a certain number of players. The nature ofthe game will then dictate limitations on the number of verbalcommunication channels required. For example, a game such as chess willnormally be played using the point-to-point scenario, because chesstypically involves only two players. The voice communicationfunctionality enables the two players to talk to each other whileplaying a chess game. For this point-to-point scenario, each gameconsole would need to instantiate only one encoder and one decoder,since more encoders and decoders are not required. During each voiceframe update, the CPU on a game console will update any encoding anddecoding as necessary. Using a predefined encode CPU usage limit of 1.5percent and a decode CPU usage limit of 0.5 percent in thepoint-to-point scenario, the total requirement for CPU usage would beonly about 2.0 percent.

As shown in the functional block diagram of FIG. 4, a game console 102is coupled in voice communication with a game console 172. Microphone144 responds to the voice of the player using console 102, and the voicecommunication module connected to the microphone produces pulse codemodulated (PCM) data that are input to a single stream encoder 160. Inresponse to the PCM data, the encoder produces compressed data packetsthat are then transmitted over a network 170 to which game console 172is connected.

Alternatively, signal stream encoding can be carried out by the DSP ofvoice communication module. In this embodiment, microphone 144 respondsto the voice of the player using game console 102, and DSP 156 isconnected to ADC 158 and produces the compressed data packets that arethen sent to game console 102 for transmission over network 170 to gameconsole 172.

The compressed data from game console 102 are input to a network queue174 in game console 102. The purpose of using a network queue to receivesound packet compressed data from console 102 is to remove jitter andother timing anomalies that occur when data are sent over network 170.The output of the single stream decoder is PCM data which are thenapplied to the DAC in the voice communication module of the player usinggame console 172 to produce an analog output that drives a headphone178.

In an alternative embodiment, the compressed data are conveyed from gameconsole 102 to DSP 156 in the voice communication module. The DSPdecodes the compressed data, converting to a corresponding PCM signal,which is applied to DAC 161 in the voice communication module of theplayer using game console 172, to produce a corresponding analog outputsignal used to drive headphone 178.

Similarly, for verbal communications from the player using console 172,a microphone 180 converts the sound incident on it into PCM data usingthe ADC within the communication module to which microphone 180 isconnected, and the PCM data are input to a single stream encoder 182,which produces compressed data that are conveyed through network 170 toa network queue 162 within game console 102. The compressed data fromnetwork queue 162 are input to a single stream decoder 168, whichproduces PCM data that are input to DAC converter in the voicecommunication module to which headphone 146 is connected. The DACproduces a corresponding analog sound signal. Thus, headphone 146receives the analog sound signal corresponding to the sound of theplayer connected to console 172 (with any voice effects added).

In the multipoint scenario, where there is one player on each console,but multiple game consoles participating in a game session, the gamedesigner can determine if all players should be able to verballycommunicate with all of the other players playing the game, or if therewill be teams comprising subsets of the players, so that only theplayers in on the same team may talk to each other. For example, if thegame being played in multipoint scenario is a card game, there might befour individual players, one each per game console, or there might betwo teams of two players each. If there are four separate players, eachgame console would instantiate one encoder and one four-to-one decoder(as discussed below). During a voice frame update, each console wouldupdate any encoding necessary for transmitting speech by the singleplayer using the game console, and decoding of speech data from any ofthe (up to) three other players on the other game consoles participatingin the game. For this scenario, using a predefined encode limit for CPUusage of 1.5 percent and a four-to-one decoder limit for CPU usage ofabout 1.3 percent, the total would be about 2.8 percent CPU usage on anyof the four game consoles being used to play the card game.

FIG. 3 illustrates how the multipoint scenario is functionallyimplemented on each game console, but does not show the other gameconsoles. However, it will be understood that network 170 couples theother game consoles in communication with the illustrated game console.As noted above, the game console includes a single stream encoder 160,which receives the PCM data produced by the ADC in the voicecommunication module (not shown in FIG. 3) of the player. The PCM dataare input into single stream encoder 160, producing voice frames ofcompressed data in packet form for transmission over network 170 to theother game consoles. Similarly, packets of compressed data are conveyedthrough network 170 from the other game consoles to the illustrated gameconsole. Each player participating in the game (or channel) has anetwork queue on the game console in which the data packets aretemporarily stored, to ensure that jitter and other timing problems areminimized when the packets of compressed data are selected by aselection engine 164 for mixing by a mixer 166 and decoding by decoder168. Decoder 168 produces PCM data that are supplied to the DAC, whichproduces the analog signal that drives headphone 146.

In an alternative embodiment, selection engine 164 conveys the twoselected compressed data streams to DSP 156 of the voice communicationmodule (shown in FIG. 2) for mixing and decompression. DSP 156 producesa corresponding PCM signal that is supplied to the DAC in the voicecommunication module, which in turn, produces the corresponding analogsignal that drives headphone 146.

As indicated in FIG. 3, network queues 162 a, 162 b, and 162 c arerespectively provided for each of the other players—up through Nplayers.

In the multipoint scenario discussed above, there are only three networkqueues, since there are only three other players engaged in the cardgame. Since mixer 166 only combines two inputs at a time, decoder 168only can provide simultaneous PCM data for two players at a time to theplayer wearing headphone 146. In contrast, an alternative is also shownin which a decoder 168′ includes a mixer 166′ that combines four datapackets from a selection engine 164′ at a time, to produce the outputprovided to headphone 146. In this alternative, the player is providedup to four other voice data packets simultaneously.

Alternatively, selection engine 164 can be employed to convey fourselected compressed data streams to DSP 156 in the voice communicationmodule of an intended recipient. DSP 156 again produces a correspondingPCM signal that is supplied to the DAC in the voice communicationmodule, producing the corresponding analog signal to drive headphone146.

Functional details for the “multiplayer on game console” scenario areillustrated in FIG. 5 where game console 102 is connected to network 170through a network layer 206 and thus to a game console 210, havingplayers 216 a and 216 b, to a game console 212 having a single player218, and to a game console 214 having four players 220 a, 220 b, 220 c,and 220 d. Game console 102 has players 200 a, 200 b, 200 c, and 200 dconnected thereto, and each player is provided with a game controllerhaving a voice communication module. For this scenario, all of theplayers on each of consoles 210, 212, and 214 have voice communicationsthat are encoded to produce single streams of compressed data packets.Network layer 206 in game console 102 conveys the data packets from eachof the three other game consoles into three separate network queues,including a network queue 162 for second game console 210, a networkqueue 184 for third game console 212, and a network queue 186 for fourthgame console 214. The output from the three network queues in gameconsole 102 is input to decoder 168, and its output is applied to anoutput router 188 that determines the specific headphone that receivesvoice communications 190 a through 190 d.

An alternative embodiment employs output router 188 to bypass decoder168 and pulls compressed data packets directly from network queues 162,184, and 186. Output router 188 conveys the compressed data to the DSPin the voice communication module of the intended recipient, so that theheadphone of that player receives voice communications 190 a through 190d.

Accordingly, each of players 200 a through 200 d receives only the voicecommunications intended for that player. Similarly, each of the fourplayers on game console 102 has a sound input from their correspondingmicrophones 202 a, 202 b, 202 c, and 202 d supplied to an input router204, which selectively applies the PCM data streams to encoder 160,which has an output coupled to network layer 206. The network layerensures that the compressed data packets conveying sound data aretransported over network 170 to the appropriate one of game consoles210, 212, and 214. The output router in each game console with multipleplayers determines the player(s) who will receive the voicecommunication from a player using game console 102.

Another embodiment bypasses input router 204 and encoder 160 by encodingthe compressed data using the DSP in the voice communication module ofthe player who is speaking.

Prioritization/Round Robin Technique for Encoding

FIG. 6 illustrates functional aspects regarding prioritization on a gameconsole handling voice communication by up to four players who are usingthe game console to play a game. During each voice interval, only two offour encoder instances are active, so that there are fewer encoders thanthere are players having voice communication capability, on the gameconsole. Thus, although there are four microphone 211 a, 211 b, 211 c,and 211 d, the digital PCM data from the ADCs in the voice communicationmodules respectively connected to the microphones are not all encoded atthe same time. Each stream of PCM data is applied to a voice activationdetection algorithm, as indicated in blocks 213 a, 213 b, 213 c, and 213d. This algorithm determines when a player is speaking into themicrophone and producing PCM data that should be encoded fortransmission to one or more other players in a game. In the worse casescenario, all four players might be speaking at the same time so thatthe voice activation detection algorithm would indicate that PCM datafrom all four microphones connected to the game console need to beencoded. However, since only two voice streams can be encoded at onetime in this preferred embodiment, a prioritizing algorithm in a block215 determines or selects the streams of PCM data that are input to thetwo parallel encoders 160′. These encoders produce compressed data inpacketized frames that are conveyed over the network (assuming that thegame console is connected to one or more other game consoles over a linkor network). In the example shown in FIG. 6, the prioritizing algorithmhas selected two streams of PCM data, including PCM data 217 c and 217d, as having the highest priority for encoding in the current frame foroutput over the network. In contrast, PCM data in streams 217 a and 217b are marked as not having a voice input, but will have the highestpriority for encoding in the next frame of compressed data if they theninclude voice data.

A round-robin encoding method is used to enable two parallel encoders toencode voice communications from four players, so that fewer encodersare required on the game console than the total number of players thatmay be speaking during any given voice data frame. FIG. 12 providesdetails concerning the logical steps that are implemented to enableround-robin encoding of a voice frame, beginning with a start block 380.In a step 382, an array of the four PCM packets (one for each player) isassembled. In the case where a player does not have a voicecommunicator, a PCM silence packet is inserted into the array. Adecision step 384 determines if the logic is carrying out a first loopfor the current voice frame. If so, a step 386 provides for preparingpriorities in an array using the variable priority (i) where i can havethe values 0, 1, 2, or 3. Thereafter, the logic proceeds with a step390.

If the logic is not making a first loop for the current voice frame, itproceeds to a step 388, wherein the logic uses the priorities array thatwas generated in a previous loop for the current voice frame.Thereafter, the logic also proceeds to step 390. In step 390, thedetection of voice activation is carried out so that PCM packets aremarked to indicate whether they have voice content. The algorithmdetects whether the current sound level is substantially greater than anaverage (background) level, which indicates that a player with amicrophone is probably currently speaking into it. Alternatively, thePCM packets can be analyzed to detect voice characteristics, whichdiffer substantially from background noise characteristics. A step 392then initializes the variable priorities index as being equal to zeroand a variable encodeops as being equal to zero.

A decision step 394 determines if the priorities index variable is lessthan four and whether the encodeops variable is less than two. Sincedecision step 394 is initially reached immediately after step 392 inwhich these two variables have been initialized to zero, both thesecriteria are met, leading to a step 402. In step 402, a variable PCMstream index is set equal to the variable priorities with a value iequal to the priorities index variable. In the initial pass for a voiceframe, the PCMstream index variable is set equal to priorities [0].

A decision step 404 then determines if voice has been detected for PCMpacket with an index equal to the PCM stream index. Again, with theinitial pass through this logic during a voice frame, the decision stepdetermines if a voice was detected for PCMpacket [PCM stream index]. Ifso, a step 406 moves the variable priorities [priorities index], whichis at the end of the priorities array and shifts all other elementsafter it one place forward. A step 408 then sets the variable encodeopsequal to its previous value plus one, thereby incrementing the variable.If voice was not detected in decision step 404, a step 410 setspriorities index equal to priorities index plus one, therebyincrementing that variable. Following either step 408 or step 410, thelogic proceeds with decision step 394.

Once the priorities index variable is equal to four or the encodeopsvariable is equal to two, the logic proceeds to a step 396. In thisstep, the logic sets the voice detected property for PCM packets[priorities [0]] and PCM packets [priorities [1]] on false. A step 398then provides for parallel encoding of PCM packet [priorities [2]] andPCM packet [priorities [3]]. Finally, in a step 400, the logic assemblesan array of four compressed data packets for transmission over thenetwork for the current voice frame. Based upon this logic, it will beapparent that if all four players are actually speaking, PCM packetswill be encoded to form the compressed packets using this round-robinalgorithm so that all of the players on a voice console can communicatewith other players in the game.

It may be helpful to work through an example in which it is assumed thatplayers one, two, three, and four are all talking at the same time. Ahistory of the last two voices or players that have been encoded ismaintained. The logic starts looking at the current microphone packetfor player one. If a voice is detected by the algorithm, it is encoded.Next, the same determination is made for player two, i.e., if a voice ispresent at player two's microphone, it is encoded in the current voiceframe. The initial history starts out with the players ordered [1, 2, 3,4], but at this point, it is updated so that the order is players [3, 4,1, 2]. The logic loops back, after a predefined microphone encodinginterval, to process the audio data for the two players that were notprocessed the last time. Currently the history list is [3, 4, 1, 2], soa check is made to determine if player three currently has voice inputon his microphone, and if so, it is encoded. However, if player three isno longer talking at this time, the logic instead proceeds to playerfour, who it is assumed is talking. Accordingly, the digital PCM voicepacket for player four is encoded and the history is updated to [3, 1,2, 4]. Next, the logic proceeds to player one, encoding that player'svoice, producing a new history [3, 2, 4, 1]. The logic will then startwith players three and two. Assuming that player three still is notspeaking so that there is no voice at that player's microphone, thelogic encodes the digital PCM packets for players two and four, yieldinga history list [3, 1, 2, 4].

In one embodiment, for each PCM packet of a player that is skipped andnot encoded, the previous packet for that player is attenuated andreplayed for the voice frame. In the worst possible state, when allplayers are talking and there are actually four different players on thegame console who are in different teams, every other PCM packet of eachplayer is skipped. Although this approach may have a slight negativeimpact on the quality of the voice of each player, it is the worst casescenario, and this scenario typically occurs infrequently during gameplay.

It should be noted that PCM packets of a player that are skipped andthus must be filled in by repeating the previous packet are nottransmitted over the network. Instead, the repeated PCM packet ishandled by the receiving game console used by the intended recipient ofthe packet. Accordingly, at most, two packets are sent from a gameconsole during any one voice frame, instead of the maximum of four. Thequeue buffers the previous packet and provides it to replace a skippedpacket. Alternatively, a skipped packet will not be put into the queueby the receiving game console, but instead, a notification indicatingthat the packet was skipped by the game console that made thetransmission will be inserted into the network queue of the receivinggame console, for that channel.

The round-robin encoding technique only operates on two frames of speechat a time and repeats the other frames of those streams that are notcurrently encoded. As noted above, this can result in degradation ofsound when all four players on a game console are speaking, but thetechnique avoids using additional CPU resources to separately encode thevoices of all four players, which might have a negative impact on gameplay.

In an alternative embodiment, one encoder per player is allocated forencoding speech. However, this embodiment is less desirable, because itrequires twice the computational resources as the embodiment discussedabove.

Voice Communication over Link/Network

FIG. 10 illustrates the general steps applied for enabling voicecommunications over a link or network. Initially, a game is started in astep 330. Next, a decision step 332 determines if the player is stoppedfrom communicating by voice with other players. This state may resultbecause of the player being banned or suspended from voice communicationby the online game service due to violations of a code of conduct orother service policies. Another reason voice may be blocked is due to adetermination by an authorized person such as a parent that a minorchild should not be permitted to engage in voice communications withother players during a game. This option is available and can be setusing options provided on the game console for specific player accounts.Once set, the data are stored on the online game service, and blockageof voice communication is enforced each time the player connects to theservice. If the current player's voice communication is blocked, a step334 determines that the game console need not process voicecommunications and instead, proceeds to a next speaker, in a step 336.Assuming that the next speaker is not precluded from having voicecommunication by a setting on the game console, the logic advances to astep 338, which gets the PCM data from the ADC in the voicecommunication module for that player. Next, the logic compresses the PCMspeech data into compressed data in a step 340. A step 342 applies anyassigned voice effects when compressing the current player's PCM speechdata to alter the characteristics of the player's voice. In a step 344,the compressed data are transmitted over a network 346 to an intendedreceiving game console to reach the intended recipients that have avoice communication module. A step 348 provides for processing the nextspeaker on the game console that is transmitting compressed data,thereby returning to decision step 332.

On the game console that has received the voice communication overnetwork 346, a step 352 provides for adding the compressed data to anetwork queue of such data. Next, in a step 354, the game consoledecompresses the compressed data pulled from the queue, producingcorresponding PCM data. A step 356 provides for adding any optionalenvironmental effects. Such effects are generally determined by optionsprovided in a game being played on the game console. For example, anenvironmental effect might include adding an echo, or introducing areverberation if the environment of the game is within a cavern, or theenvironmental effect might involve providing a frequency bandequalization, e.g., by adding a bass boost for play of the audio data onsmall speakers. Next, a step 358 mixes voice streams received from aplurality of different players into one output voice stream that will beprovided to an intended recipient player. The output voice stream isconveyed as PCM data to the DAC associated with the headphone of theintended recipient player in a step 360, which produces a correspondinganalog signal to drive the headphone. The player thus hears the voicecommunication from each of the players that were decoded and mixed intothe output voice steam.

Muted Voice Communication between Players

Another situation arises whenever a specific player has been muted fromvoice communication by another player. Once a player has thus beenmuted, the specific muted player will be unable to either hear or speakwith the muting player. The muting player must explicitly unmute themuted player to restore voice communication.

Handling of Network Data Packets

FIG. 11 illustrates further details in regard to the receipt of voicecommunication data as packets over network 346. As indicated in block351 a and 351 b, compressed data are received over the network from Nother game consoles. Each channel of the compressed data is initiallyinput into one of N queues 351 a–351 b, where a separate queue isprovided for each player on a connected game console (one or moreplayers on each game console). A block 364 then synchronizes the Nqueues that have been formed to receive the compressed data. In thisstep, the encoded and compressed compressed packets are obtained fromall of the queues for a current voice frame. A selection engine 366 thendetermines the compressed packets that should be assembled for input tothe decoding engine in a block 368. The decoding engine decompresses anddecodes the compressed data, converting the data into PCM data that arethen applied to each of the connected voice peripheral communicationsmodules 370 through 372 to enable the players respectively using thosevoice communication modules to hear the sound that was transmitted overthe network to them.

Details relating to the processing of encoded packets that are receivedfrom each queue are shown in FIG. 7. In a block 221, the CPU checks eachqueue to obtain an encoded compressed packet or the queue notifies theclient CPU that it does not have a packet from a player, but that apacket for that player should have been in the queue. In this case, theCPU determines if the previous packet obtained for that player from thequeue was in fact a valid encoded packet and if so, the CPU copies theprevious packet for that player so it can be provided for furtherprocessing in the next block. However, if the previous packet for thatplayer was also missing, the CPU does an attenuation on previous packetsfor that player. The purpose of this step is to minimize the perceptionof silence caused by a missing packet resulting from skipping packetsduring round-robin encoding and from dropped packets due to networkconditions.

Next, in a block 222, the packets that have been obtained in block 221are ordered, and all silent packets in the order are eliminated. Theresult is a subset of the packets originally provided in the queues. Dueto the processing noted above, all packets in the subset will containvalid voice data.

Next, a block 224 provides for applying channel masks to the voice data.Each player is associated with a channel, and all players on aparticular channel are able to communicate by voice with each other. Forexample, one channel may be used for voice communication between aplayer who is designated as a team leader and team members, enablingthat player to communicate with all members of the team. In addition,the team leader may also be able to select another channel for verbalcommunication with another player designated as a commander, who is ableto communicate with a plurality of team leaders, or yet another channelto enable the team leader to communicate only with the other teamleaders. In this implementation, each player who is talking is given a16-bit word that defines the “talker channel” for the player. The gamedetermines what the individual bits of the word for the talker channelmean, e.g., indicating that the talker channel is for a team, a teamleader, a commander, etc. In addition, each player can be assigned a“listener channel” on which they can receive speech. When a voicecommunication comes in over the network, the talker channel is logically“ANDed” with the listener channel for a given player, and if the resultis not zero, then that player is able to hear the voice communication.In this manner, the game being played (or each player, within theconstraints of the game) is able to select arbitrary permutations thatdetermine the players that are coupled in voice communication with otherplayers.

Referring again to FIG. 7, block 224 enables each player to choose tolisten to only some channels and to employ a channel mask. A channelmask is applied for each player on the game console resulting in up tofour subsets of voice streams—one subset for each player listening,based upon the player's individual listener mask. Each person who islistening will have a list of candidate packets included in differentvoice streams.

In a block 226, the muting mask and user defined priorities are applied.While a preferred embodiment enables a player to selectively precludefurther voice communications with a selected player, it is alsocontemplated that a player might selectively only mute voicecommunications with a specific player during a current game session. Inthis alternative embodiment, each player on a game console might chooseto mute certain people on listening channels. Voice streams from playerswho have been muted by that player will then be eliminated from the listof candidate voice streams for that player on the game console. Anyremaining voice streams for each player are sorted by user-definedpriorities. In this step, one voice stream may have the highest priorityall the time. For example, the game (or a player—if the game permitsthis option) may selectively set a channel coupling team member in voicecommunication with a team leader so that that channel has the highestpriority for each of the team members. Loudness of the incoming voicestream can also be a basis for determining the priority of a channel fora given player, so that the player hears the loudest voice streams allof the time. Alternatively, or in addition, if a voice stream hasstarted to render, the game will wait for it to finish regardless of theloudness of the other voice streams that started later so that asentence is not cut off in “mid-stream,” before it is finished. As afurther alternative or in addition, other priorities can be defined foreach voice channel. For example, a game-specific role related prioritycan be applied.

Following block 226, decoding is applied to the voice streams resultingfrom applying the muting masks and user/game defined priorities usingeither a decoding engine type one as indicated in a block 228 or adecoding engine type two as indicated in a block 230. In block 228,decoding engine type one allocates decoders, mixing and decoding foreach player method. In this algorithm, for each player, the first Npackets in a list of ordered packets are selected. If the list containsless than N elements, silent compressed data packets used instead of theinexistent packets to avoid producing spikes in the CPU processing ofthe voice packets.

In block 230, when applying decoding using engine type two, decoders areallocated for decoding and mixing in a DSP method. In accordance withthis algorithm, until the maximum number of decoded packets is reached,or the end of the candidate packet is reach, or the end of the candidatepacket list is reached, the current player is obtained from the orderedlist of packets, if the list is not empty. If the head of the list ofordered packets has not been chosen before, the head of the list is thenchosen to be decoded and the counter is incremented for the decodedpackets. Thereafter, the head of the list is eliminated from the orderedlist. Next, the algorithm moves to the next player, who then becomes thecurrent player. For example, after player four, player one would thenagain become the current player. If any decoding slots remain fordecoding additional voice packets, silence packets are applied to theparallel decoder to avoid spikes in CPU processing of the voice packets.

Decoding Engines, Types One and Two

In FIG. 8, details relating to the functional aspects of decoding enginetype one are illustrated. As shown therein, encoded streams from onethrough N designated by reference numerals 240, 242, and 244 providecompressed data to a selection engine 257 that chooses two encodedstreams for decoding for each player headphone. In this case, an encodedstream 1.1 and an encoded stream 1.2 are selected for input to decoder168 where the streams are mixed by a mixer 252 prior to decoding. Theoutput of the decoder is PCM data that are supplied to the DAC withinthe voice communication module for a headset 248. Similarly, for each ofthe other player headphones, another decoder 168 receives encoded voicestreams as compressed data, which are then mixed and decoded. As shown,the decoder for a fourth player includes a mixer 254 that mixes encodedvoice streams 4.1 and 4.2 so that the decoder produces PCM data that aresupplied to the DAC that provides the analog signal to drive a headphone250 for the fourth player on the voice console. Of course there areinstances where less than four players will be using a game console, inwhich case, fewer than four decoders are required to be active.

An alternative embodiment to both decoding engine type 1 and decodingengine type 2 conveys the prioritized voice streams (at block 226)directly to the DSP in the voice communication module of the intendedrecipient.

Functional aspects of the type two decoding engine are illustrated inFIG. 9A. Again, encoded streams one through N represented by referencenumbers 240, 242, and 244 are input to a selection engine 260, whichchooses the maximum four encoder streams, a minimum of one for eachplayer who is listening and has a voice stream addressed to that player.In this case, four parallel decoders 262 receive the four selectedencoded voice streams of compressed data. The decoders then decode thecompressed data and supply the resulting PCM data to a mixer for eachplayer to whom any voice stream was intended. In this case, the firstplayer receives two voice streams from other players that are mixed by amixer 270 in a mixer identified by reference number 264. Although notshown, each of the other players receiving voice communications fromother players in the game would also be provided with a separate mixingbin to which the output of the four parallel decoders is applied formixing. For example, the fourth player receives three voice streams thatare mixed by a mixer 272 in a mixer identified by reference numeral 266.The resulting PCM data are then applied to headset 250 for the fourthplayer. Thus, each player can receive from one to four voice streamsthat are mixed by the four-in-one mixing bin assigned to that player.

Another embodiment (at block 260) bypasses decoder 262 and mixers 264and 266 and conveys the compressed data to the DSPs of the voicecommunication modules coupled to headphones 248 and 250, respectively.

An alternative approach for use in the type two decoding engine isillustrated in FIG. 9B. In this embodiment, the relative disposition ofthe decoder and the mixers are reversed from that shown in FIG. 9A.Specifically, each player is provided with a two-stream mixer that iscoupled to receive and mix the compressed data incoming over thenetwork. A two-stream mixer 280 is provided for player one, a two-streammixer 282 for player two, a two-stream mixer 284 for player three, and atwo-stream mixer 286 is provided for player four. Up to two voicestreams of compressed data are thus input to each of these mixers perplayer and are mixed, providing a single mixed stream from each mixerfor input to a four-stream parallel decoder 288. This decoder thendecodes the compressed data producing PCM data that is supplied to eachplayer headphone 248, 252, 253, and 250 who is an intended recipient ofa voice communication from another player.

Yet another embodiment provides that the compressed data conveyed totwo-steam mixers 28, 282, 284, and 286 is decoded by the DSPs of thevoice communication modules coupled respectively to headphones 248, 252,253, and 250.

Controlling Voice Communication with Other Players

As noted above, a player has the option of precluding further voicecommunications with a specific player because of behavioral issues orfor other reasons. For example, if a specific other player tends to usesexcessive profanity, a player may choose not to engage in furthercommunications with that other player. Each game will generally providean option for a player to mute voice communications with another playerfor the current and future game sessions, and it is also contemplatedthat a player might also be enabled to mute voice communications withanother player for only a current game session. FIGS. 13, 13A, and 13Billustrate exemplary dialog boxes that enable this control of voicecommunication to be implemented in a game called “MY GAME.” Thisfictitious game lists each of the players, as shown in a player list box430. Player list box 430 includes six players of which the top listedplayer has been selected as indicated by a selection bar 434. Thisplayer, who plays the game using the alias “Avenger,” has voicecommunications capability, as indicated by a speaker symbol 436 that isshown in one column of the dialog, in the same row as the alias. Playersrespectively using the aliases “Iceman” and “Raceox” do not have voicecommunication capability, as is evident by the lack of speaker symbol436 in this column, in the same row as either of these aliases. A radiobutton 438 is provided to enable any of the players listed to be mutedfrom voice communication with the current player who is viewing playerslist box 430. In this case, Avenger has been selected by the player, asindicated by radio button 438. When the player is selected, a window 440opens that identifies the selected player and notes that this player iscurrently one of the participants in the game and has a voicecommunication module. If the player viewing player's list box 430 clickson a select button 442, a voice communication status select 450 opensthat includes an option bar 452, which can be toggled to differentstates. As shown in FIG. 13A, option bar 452 indicates that the selectedplayer has been enabled to verbally communicate with the player who isselecting this option. In FIG. 13B, the option bar has been toggled to astate 454, which indicates that the player viewing the state wants tomute the selected player for the current game session. Depending uponthe selection that the player makes, speaker symbol 436 will change.Instead of that shown in FIG. 13, if the player selected the option tomute the specific player has been chosen a dash box will appear aroundthe speaker symbol shown. This type of muting expires after the currentgame session ends. On the other hand if the specific player has beenselectively locked out, a dash heavy-bar box will be added aroundspeaker symbol 436. In this case, the decision to mute a player can onlybe turned off by the player making that decision from within a gamesession or using a system control for the game console. Yet anothersymbol (not shown) is used to indicate that voice communications forcorresponding player are being played through loudspeakers (e.g.,television or monitor speaker) connected to the game console of therecipient player, rather than through a headphone. This option can beselected by a player, who prefers not to wear a headset, but is lessdesirable, since the player will not be using a microphone to verballycommunicate with other players.

In the event that a number of players provide negative feedbackconcerning a specific player based upon the verbal behavior of thatplayer being deemed to be unacceptable, such as excessive use ofprofanity, or use of sexually explicit language, the online game servicecan automatically determine that the number of complaints received hasexceeded a threshold, causing the specific player to be banned fromfurther voice communication. The ban might initially be for a limitedperiod of time such as a week, and then subsequently, if furthercomplaints are received beyond the threshold, the specific player mightbe banned permanently from voice communication. A specific player thathas been banned in this manner will be informed first of the temporarysuspension of voice communication capability, and then of the permanentsuspension, if the behavior of the player causes that result. Each timea player logs into the online game service on a game console, permissionflags are downloaded to the game console as part of the sign in process.These flags include information about various aspects of the system. Oneof the flags determines whether a specific player has permission toengage in voice chat. Accordingly, in the event that a specific playerviolates the terms of service or code of conduct, the bit controllingthe ability of the player to communicate by voice can be changed topreclude such voice communications.

Once a player has elected to preclude voice communications with aspecific player, the identification of the specific player is preferablytransmitted to an online game service and stored there in relation tothe identity of the player making that election, so that in futuresessions of any games, the player who has made such a decision will notreceive any voice communication from the specific other player and willnot transmit any voice communication to the specific other player. Thisdecision will not be apparent to the specific other player, since thedialog box showing the status of players in a game will simply displayan indication on the specific other player's view that the player makingthat decision lacks voice communication capability, and in the dialogdisplayed to the player making the decision, the muted status of thespecific other player will be indicated. Thus, even though the specificplayer changes the alias used or signs on with a different game console,the prohibition against voice communication for the specific player madeby a player will continue in force.

It is also contemplated that the PCM data will also include lip positioninformation associated with each segment of speech. The lip-syncinformation will be generated and transmitted to the encoder with thePCM data, converted to compressed data, and transmitted to recipientplayer so that when a player is speaking, the character represented andcontrolled by that player in the game appears to be speaking insynchronization with the words spoken by the player. Depending upon thenature of the graphics character representing the player who isspeaking, the nature of the “mouth” may differ from that of a normaloral portion of a human's anatomy. For example, a character in a gamemight be an alien that has mandibles that move when the characterspeaks. Nevertheless, the synchronization of the oral portion of thecharacter with the word spoken adds to the realism in game play. Detailsfor accomplishing lip-sync using graphic characters and spoken words aredisclosed in a commonly assigned U.S. Pat. No. 6,067,095 the disclosureand drawings of which are hereby specifically incorporated herein byreference. Alternatively, lip synchronization information can beextracted from the compressed data during decoding.

One of the advantages of the present invention is that it combines voicestreams for all of the players on a game console into a singlecompressed data stream to more efficiently transmit data over a networkwithin a limited bandwidth. Thus, when players on the same game consoleare talking to all of the other players participating in a game, all ofthe voice data for the players on the same console are combined into onenetwork stream. It is not necessary to send multiple voice data streamsfrom the game console.

Another advantage of the present invention is its ability to allocate amaximum number of encoders that is half of the number of players thatmight be playing a game on a game console. Accordingly, the gamedesigner can determine the amount of resources to be allocated to voicecommunications and can limit those resources by, for example, providingonly two encoders and requiring that the encoders operate in round-robinas discussed above. Although there is a slight negative effect fromusing previously transmitted packets when carrying out the round-robinapproach, the adverse effect on the quality of voice communication isgreatly outweighed by the limitation on the use of computing resourcesfor voice communications to minimize adverse effects on the quality ofgame play.

When participating in a game over the Internet or other network, aplayer may optionally, depending upon the game being played, choose toplay only with players who agree to a specific language in which voicecommunications are to be conducted. Also, the player can optionallydetermine that the game will only be played with those players havingvoice communication capability. Similarly, players without voicecommunication capability may selectively engage join in games only withthose other players who also do not have voice communication capability.

Although the present invention has been described in connection with thepreferred form of practicing it, those of ordinary skill in the art willunderstand that many modifications can be made thereto within the scopeof the claims that follow. Accordingly, it is not intended that thescope of the invention in any way be limited by the above description,but instead be determined entirely by reference to the claims thatfollow.

1. A method for encoding a plurality of audio channels during play of anelectronic game, using at least one encoder, but fewer encoders thanaudio channels to be encoded, each encoder encoding audio signals in anactive audio channel and producing corresponding data packets fortransmission to at least one recipient participating in the play of theelectronic game, comprising the steps of: (a) creating a round robinhistory of the audio channels that have been encoded by the encodersduring successive processing intervals; (b) if more than one audiochannel per encoder are simultaneously active, selecting an audiochannel to encode with each encoder based upon the round robin historyand on the audio channels that are then active, so that an audio channelthat is active will be skipped and not encoded more frequently thanevery other processing interval; and (c) updating the round robinhistory for each processing interval to indicate each audio channel thatwas encoded during the processing interval.
 2. The method of claim 1,wherein the step of creating the round robin history comprises the stepof indicating each audio channel that has just been encoded during acurrent processing interval, at a lower priority position in the roundrobin history.
 3. The method of claim 2, wherein the round robin historyincludes indications of the audio channels in successively lowerpriority positions, from a highest priority to the lowest priorityposition, and wherein the step of selecting the audio channel to encodecomprises the step of checking the audio channels that are indicated insaid positions, starting at the highest priority position in the roundrobin history to determine if the audio channel indicated is active andif so, encoding said audio channel with any available encoder, so thataudio channels that are active and have a higher priority position inthe round robin history list are selected first for encoding.
 4. Themethod of claim 1, wherein if a data packet for a specific audio channelthat is still active was not encoded in a processing interval, furthercomprising the step of duplicating audio data from a data packet thatwas encoded and transmitted for said specific audio channel during animmediately prior processing interval, to provide audio data to beplayed for said specific audio channel for the processing interval inwhich the audio data packet was not encoded.
 5. The method of claim 4,wherein the step of duplicating the audio data is carried out by arecipient of the data packet for the specific audio channel, whenplaying the audio data for the specific audio channel.
 6. The method ofclaim 5, wherein the duplicate audio packet for the specific audiochannel is not transmitted to any recipient.
 7. The method of claim 1,wherein there are up to two encoders and up to four audio channels to beencoded, each encoder encoding no more than one data packet during eachprocessing interval.
 8. A memory medium having machine executableinstructions for carrying out the steps of claim
 1. 9. A system forencoding a plurality of audio channels during play of an electronicgame, comprising: (a) a processor; (b) a memory that is coupled to theprocessor, said memory storing machine instructions for causing theprocessor to carry out a plurality of functions, said functionsincluding executing an instance of an electronic game; (c) audio inputtransducers and audio output transducers, coupled to the processor andemployed for input and output of audio signals, each audio inputtransducer providing an audio signal to a corresponding audio channel tobe encoded and transmitted to at least one recipient, each audio outputtransducer being employed to play an audio signal received from anotherplayer participating in the play of an electronic game; (d) at least oneencoder, but fewer encoders than audio channels to be encoded, eachencoder encoding audio signals in an active audio channel and producingcorresponding data packets for transmission to at least one recipientparticipating in the play of an electronic game, wherein said machineinstruction cause the processor to carry out a plurality of functions,including: (i) creating a round robin history of the audio channels thathave been encoded by the encoders during successive processingintervals; (ii) if more than one audio channel per encoder aresimultaneously active, selecting an audio channel to encode with eachencoder based upon the round robin history and on the audio channelsthat are then active, so that an audio channel that is active will beskipped and not encoded more frequently than every other processinginterval; and (iii) updating the round robin history for each processinginterval to indicate each audio channel that was encoded during theprocessing interval.
 10. The system of claim 9, wherein the machineinstructions further cause the processor to indicate each audio channelthat has just been encoded during a current processing interval, at alower priority position in the round robin history.
 11. The system ofclaim 10, wherein the round robin history includes indications of theaudio channels in successively lower priority positions, from a highestpriority to the lowest priority position, and wherein the machineinstructions cause the processor to check the audio channels that areindicated in successive positions, starting at the highest priorityposition in the round robin history to determine if the audio channelindicated is active and if so, encoding said audio channel with anyavailable encoder, so that audio channels that are active and have ahigher priority position in the round robin history list are selectedfirst for encoding.
 12. The system of claim 9, wherein there are up totwo encoders and up to four audio channels to be encoded, each encoderencoding no more than one data packet during each processing interval.13. A method for decoding audio channels during play of an electronicgame, to produce an audible audio output for at least one playerintended as a recipient of audio content from another playerparticipating in playing the electronic game, comprising the steps of:(a) instantiating a selected number of decoders for decoding datapackets received, to recover the audio content conveyed by the datapackets, wherein each decoder is associated with a different listeningchannel; (b) creating queues of data packets received from audio sourcesin different listening channels, each listening channel including datapackets from one or more sources for decoding by the decoder associatedwith the listening channel; (c) if a data packet for an audio source ismissing in a stream of data packets in a listening channel, replicatinga content of a previous data packet for said source, for use instead ofthe data packet that was missing; (d) selecting data packets from thequeues to be decoded, but not exceeding the number of decodersavailable; and (e) decoding the data packets that were selected, usingthe decoders that were instantiated.
 14. The method of claim 13, whereinthe step of selecting comprises the step of enabling each player to maskany listening channel to which the player does not want to listen. 15.The method of claim 13, wherein the step of selecting comprises the stepof enabling each player to selectively mute a specific source havingdata packets in a queue, so that the data packets from the specificsource are not selected for decoding for said player.
 16. The method ofclaim 13, wherein the step of selecting comprises the step of sortingthe data packets that will be decoded for each player according to anypriority assigned by the player relative to a source of the datapackets.
 17. The method of claim 13, wherein the step of selectingincludes the step of applying a priority defined by the electronic gamethat is being played, in determining the data packets included in eachlistening channel to be heard by a player when decoded.
 18. The methodof claim 13, further comprising the step of ordering the data packets asa function of a volume of the content of the data packets, wherein thestep of selecting comprises the step of eliminating each data packet inthe queues as thus ordered, that includes a substantially silentcontent.
 19. The method of claim 18, wherein for each player that willlisten to the data packets that are decoded, the step of decodingcomprises the step of decoding a first predetermined number of the datapackets that were ordered, using silent compressed data for the contentthat will be heard in place of any inexistent data packets.
 20. Themethod of claim 13, further comprising the step of mixing data packetsfrom a plurality of sources before the step of decoding.
 21. The methodof claim 13, further comprising the step of mixing content from aplurality of sources after the step of decoding.
 22. The method of claim13, further comprising the steps of: (a) ordering the data packets toproduce an ordered list for each player receiving and listening to thecontent of the data packets; (b) selecting a data packet at a head ofthe ordered list of the data packets to be decoded for a current player,if: (i) a predetermined maximum number of decoded data packets has notyet been reached; (ii) the ordered list is not empty; and (iii) the datapacket at the head of the ordered list was not chosen previously; (c)incrementing a counter of decoded data packets that is used to determineif the predetermined maximum number has been reached; (d) eliminatingthe data packet at the head of the ordered list, in each ordered list inwhich it appears; and (e) repeating the preceding three steps (b)–(d)for each successive player who will listen to the content of the datapackets.
 23. The method of claim 22, wherein the step of decoding iscarried out by a digital signal processor.
 24. A memory medium havingmachine executable instructions for carrying out the steps of claim 13.25. A system for decoding compressed data conveyed in data packets in aplurality of listening channels during play of an electronic game,comprising: (a) a processor; (b) a memory that is coupled to theprocessor, said memory storing machine instructions for causing theprocessor to carry out a plurality of functions, said functionsincluding executing an instance of an electronic game; (c) audio inputtransducers and audio output transducers, coupled to the processor andemployed for input and output of audio signals, each audio inputtransducer providing an audio signal to a corresponding audio channel tobe encoded and transmitted to at least one recipient, and each audiooutput transducer being employed to play an audio signal received from asource comprising another player participating in the play of anelectronic game; (d) a selected number of decoders used to decode thedata packets received by the system, to recover the audio contentconveyed by the data packets, wherein each decoder is associated with adifferent listening channel, wherein the machine instructions cause theprocessor to carry out a plurality functions, including; (i) creatingqueues of data packets received from audio sources in differentlistening channels, each listening channel including data packets fromone or more sources for decoding by the decoder associated with thelistening channel; (ii) if a data packet for an audio source is missingin a stream of data packets in a listening channel, replicating acontent of a previous data packet for said source, for use instead ofthe data packet that was missing; (iii) selecting data packets from thequeues to be decoded, but not exceeding the number of decodersavailable; and (iv) decoding the data packets that were selected, usingthe decoders.
 26. The system of claim 25, wherein the machineinstructions further cause the processor to enable each player to maskany listening channel to which the player does not want to listen. 27.The system of claim 25, wherein the machine instructions further causethe processor to enable each player to selectively mute a specificsource having data packets in a queue, so that the data packets from thespecific source are not selected for decoding for said player.
 28. Thesystem of claim 25, wherein the machine instructions further cause theprocessor to sort the data packets that will be decoded for each playeraccording to any priority assigned by the player relative to a source ofthe data packets.
 29. The system of claim 25, wherein the machineinstructions further cause the processor to apply a priority defined bythe electronic game that is being played, in determining the datapackets included in each listening channel to be heard by a player whendecoded.
 30. The system of claim 25, wherein the machine instructionsfurther cause the processor to order the data packets as a function of avolume of the content of the data packets, and eliminate each datapacket in the queues as thus ordered that includes a substantiallysilent content.
 31. The system of claim 30, wherein for each player thatwill listen to the data packets that are decoded, the machineinstructions cause the processor to decode a first predetermined numberof the data packets that were ordered, using silent compressed data forthe content that will be heard in place of any inexistent data packets.32. The system of claim 25, wherein the machine instructions furthercause the processor to mix the content of data packets from a pluralityof sources, before decoding the data packets.
 33. The system of claim25, wherein the machine instructions further cause the processor to mixthe content received from a plurality of sources, after decoding thedata packets.
 34. The system of claim 25, wherein the machineinstructions further cause the processor to (a) order the data packetsto produce an ordered list for each player receiving and listening tothe content of the data packets; (b) select a data packet at a head ofthe ordered list of the data packets to be decoded for a current player,if; (i) a predetermined maximum number of decoded data packets has notyet been reached; (ii) the ordered list is not empty; and (iii) the datapacket at the head of the ordered list was not chosen previously; (b)increment a counter of decoded data packets that is used to determine ifthe predetermined maximum number has been reached; (c) eliminate thedata packet at the head of the ordered list, in each ordered list inwhich it appears; and (d) repeat the preceding three steps (b)–(d) foreach successive player who will listen to the content of the datapackets.
 35. The system of claim 25, wherein each decoder comprises adigital signal processor.